Modelling Motivation as an Intrinsic Reward Signal for Reinforcement Learning Agents

Reinforcement learning agents require a learning stimulus in the form of a reward signal in order for learning to occur. Typically, this reward signal makes specific assumptions about the agent’s external environment, such as the presence of certain tasks which should be learned or the presence of a teacher to provide reward feedback. For many complex, dynamic environments, design time knowledge of the tasks to be learned, or the presence of a teacher, cannot be assumed. In order to extend reinforcement learning to such environments, this paper presents a model of motivation as an intrinsic reward signal based on the concept of events, which relaxes these assumptions. The model uses context-free grammars as an adaptable representation of environments about which there is limited design time knowledge, and events to represent potential learning tasks as changes in the agent’s environment. Within this framework, we evaluate a computational model of interest as a motivation process. This evaluation is performed in two reinforcement learning settings, flat reinforcement learning and hierarchical reinforcement learning, in terms of learning efficiency, behavioural variety and behavioural complexity. We show that motivation based on general, task-independent concepts are able to motivate learning of multiple, task-oriented behaviours in environments where neither design time knowledge of the tasks to be learned nor a teacher is available.